rpc, execution, db: FcuBackgroundCommit groundwork, coherent cache fixes - #21293

Summary

Groundwork for enabling FcuBackgroundCommit — the default flip and the standalone cache-retention increase from 0MB to 128MB are in the stacked #22269; defaults are unchanged here. With the flag on, the FCU response returns to the consensus client before the MDBX flush+commit lands: the commit runs in a background goroutine, and the ExecModule semaphore serializes successive FCUs, so FCU N+1 always reads FCU N's committed state. This PR makes RPC reads, the version-keyed state cache, and post-FCU consumers coherent under that mode. Several of the fixes matter already today, because state-change notifications are dispatched pre-commit.

RPC: head-sensitive reads resolve on a well-defined view

They see the pre-commit head everywhere or nowhere — never a mix.

Overlay view — the head and every dependent block-table read are served from the published BlockOverlay: bor getSnapshot/getAuthor/getSigners/getSnapshotProposer{,Sequence}/latest-block, eth_getBlockTransactionCountBy{Number,Hash}, graphql latest-block, debug_setHead, debug_getRawHeader, eth_getTransactionByHash, txpool_content{,From}, and erigon_getBlockByTimestamp. Behavior change on Polygon: for bor getSnapshot/getSigners/getSnapshotProposer{,Sequence}, nil/latest now resolves to the overlay-aware forkchoice/executed head instead of the header-stage tip (ReadCurrentHeader), so a catching-up node answers for its executed position rather than the downloaded-header tip — consistent with eth_blockNumber; getAuthor additionally fixes explicit-tag resolution (negative tags were cast to uint64 and returned "unknown block"). BaseAPI.headerByHash is overlay-aware, covering every by-hash consumer that stays on block tables. Each request pins one overlay read view up front and reuses it for all dependent reads, so an overlay unpublished mid-request cannot drop the request onto the older MDBX snapshot.

Committed view — the dependent reads use SD-temporal data, which the overlay does not expose, so tags resolve with nil filters and the bounds agree with the data: eth_getLogs/overlay_* range resolution (including eth_getLogs block-hash filters), trace_filter, eth_getProof, eth_simulateV1, debug_traceBlockBy*, debug_storageRangeAt, debug_accountRange, debug_accountAt (by-hash included — an overlay-resolved head would have no committed history), eth_getWitness. The filters-param contract is documented on rpchelper.GetBlockNumber. eth_getProof additionally keeps header lookup, commitment reconstruction, and state reads on the caller's single RO snapshot; shared branch-cache reads are bound-gated (servableUnderBound, #22467), so a concurrent commit cannot mix snapshots. Nil guards cover parity_listStorageKeys, trace_filter, and the eth_getProof header lookup.

Compatibility: on payload-building nodes, "pending" resolves to the latest executed block in every committed-view method that accepts the tag (eth_getLogs/overlay_*, trace_filter, debug_traceBlockByNumber, eth_simulateV1, eth_getProof, eth_getWitness). This keeps tag resolution aligned with the committed data those methods read. debug_accountRange keeps its explicit pending rejection.

Known embedded-daemon limitation: generic latest-state calls (eth_call, eth_getBalance, eth_getStorageAt, eth_getCode) resolve the overlay head while their temporal state reads stay on the committed snapshot — head N with state N-1 for the commit duration. The SD-aware temporal view needed to close this is tracked in #21314.

Coherent state cache: roots keyed by the version readers observe

db/kv/kvcache, execution/execmodule/notification_dispatcher.go. The pre-commit dispatch announces the post-commit PlainStateVersion, so version-keyed roots match exactly the readers that should see them: post-commit transactions hit the new canonical root, pre-commit transactions keep the previous one. Batch-fed storage and code entries use the key shapes readers look up (address+location; code by address). A view that outlives KeepViews version advances falls back to its own tx snapshot instead of erroring too old ViewID; reads through non-latest views are not cached (they bypass eviction accounting and would grow without bound). Retained roots share the configured memory budgets: state and code entries in older roots are cleared on version advance, and views pinned to them fall back to their own snapshots. Canonical roots start from their own batch, with no carry-over: the state-change producers don't announce every mutation, so a carried entry could stay stale — at a deliberate hit-rate cost: every version starts cold except the batch's changed keys, so hot-but-unchanged keys miss once per key per block, and steady-state hit rates are much lower than with carry-over. Completing the producers and restoring carry-over (which recovers the warmth) is tracked in #22276. The standalone daemon uses this coherent cache at every configured budget, in remote and --datadir modes alike. Positive budgets retain entries; 0MB retains none and disables new-block waiting, so reads fall back to the caller's tx snapshot while remaining coherent. The default remains 0MB until #22269 raises it to 128MB.

Engine: busy-tolerant batch execution

A background FCU commit briefly holds the exec semaphore, so ValidateChain/UpdateForkChoice can routinely return Busy. execDownloadedBatch waits it out via retryBusy: a ctx-aware 50 ms poll with a periodic debug log so a stuck commit surfaces.

Overlay safe-close invariant

db/kv/membatchwithdb. Shared-tx read views are safe only on the pure-Go memStore backing, whose Rollback/Close are no-ops on the data — that is what lets the bg-commit goroutine close the published overlay while RPC readers still iterate views. The memTx field's type restricts overlays to that backing.

Post-FCU readers wait for the commit

State-change events are dispatched pre-commit, so the notification stream alone does not guarantee MDBX contains the head. import_cmd and the execmodule tester call ExecModule.WaitIdle before opening a fresh tx after UpdateForkChoice.

Tests

TestStateChangeVersionMatchesCommitted pins announce-vs-committed version parity across {fg, bg} × {1-by-1, batched} (the batched case crosses the initial-cycle threshold, covering mid-FCU version bumps). kvcache tests pin fresh roots, retained-root aggregate budgets, reader-shaped feed keys, the evicted-view fallback, and non-latest no-cache. TestZeroBudgetRemoteCachePinsCommittedState pins snapshot-consistent reads with entry retention disabled. Overlay tests pin view selection in TestGetBlockTransactionCountByHash_SeesOverlayHead, TestDebugAccountAt_OverlayHeadHash_CommittedView, and TestGetLogsBlockHashUsesCommittedView, plus view lifetime under concurrent unpublish (the three *_PinsOverlayView tests). TestGetProofPinsReadSnapshot pins all proof reads to one RO snapshot; TestGetProofMissingHeader pins a clean error for a missing header. TestNotificationDispatchBackgroundCommit covers notification dispatch under background commit; multi-block bg-commit coverage lives in TestReorgBackAndForwardIntoCanonicalChain (bg mode) and TestInsertBlocksWithBatchedFCU_BadBlockRecovery_Background.

Safety (with the flag enabled)

FCU sequencing. The bg goroutine releases the exec semaphore only after the commit completes and PublishOverlay(nil). AssembleBlock/ValidateChain share the semaphore, so engine_newPayload/engine_getPayload serialize behind a pending commit — ~commit-duration added latency; a SYNCING response in a pipelined burst is spec-valid and self-heals.

Embedded rpcdaemon. Overlay-aware paths read FCU N's block-table writes pre-commit. In-flight readers keep their pinned view across unpublish; new readers cascade to MDBX, which by then contains N. No stale window for block-table paths; latest-state calls carry the #21314 limitation above.

Standalone rpcdaemon (remote or with datadir). With a positive entry-retention budget, cache entries are keyed by PlainStateVersion, bumped inside the commit batch and announced at the post-commit value — a daemon tx opened during the commit window resolves the N-1 root, so pre-commit batch data is never served against the older head. At 0MB, no entries are retained and each view falls back to its own tx snapshot, preserving the same consistency without serving batch data. Incomplete producer batches can still leave a retained entry stale for one version (#22276).

Interplay with #21414 (FCU semaphore decouple)

#21414 releases the exec semaphore as soon as updateForkChoice returns and moves the commit to a FIFO background worker, chaining FCU N+1's in-memory state onto FCU N's in-flight commit generation. Whichever PR lands second must re-validate the premises stated above in semaphore terms:

• "FCU N+1 reads FCU N's committed state" becomes "reads the chained in-flight generation" — TestStateChangeVersionMatchesCommitted and the cache's pre-commit-window reasoning are the regression net.
• WaitIdle currently implies the commit has landed; import_cmd and the execmodule tester rely on that.
• Overlay fallbacks rely on PublishOverlay(nil) happening only after the commit lands; the worker must preserve that ordering.
• The Busy window retryBusy waits out mostly disappears (the helper stays correct, just rarely loops).

Flag gating composes: #21414 inherits --fcu.background.commit (default false); the flip stays with #22269.